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Objectives: The hospital standardized mortality ratio (HSMR) has been widely used because it allows for robust risk adjustment using 
administrative data and is important for improving the quality of patient care. 

Methods: All inpatients discharged from hospitals with more than 700 beds (66 hospitals) in 2008 were eligible for inclusion. Using 
the claims data, 29 most responsible diagnosis (MRDx), accounting for 80% of all inpatient deaths among these hospitals, were identi- 
fied, and inpatients with those MRDx were selected. The final study population included 703 571 inpatients including 27 718 (3.9% of 
all inpatients) in-hospital deaths. Using logistic regression, risk-adjusted models for predicting in-hospital mortality were created for 
each MRDx. The HSMR of individual hospitals was calculated for each MRDx using the model coefficients. The models included age, 
gender, income level, urgency of admission, diagnosis codes, disease-specific risk factors, and comorbidities. The Elixhauser comor- 
bidity index was used to adjust for comorbidities. 

Results: For 26 out of 29 MRDx, the c-statistics of these mortality prediction models were higher than 0.8 indicating excellent discrim- 
inative power. The HSMR greatly varied across hospitals and disease groups. The academic status of the hospital was the only factor 
significantly associated with the HSMR. 

Conclusions: We found a large variation in HSMR among hospitals; therefore, efforts to reduce these variations including continuous 
monitoring and regular disclosure of the HSMR are required. 
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INTRODUCTION 

Measuring the quality of healthcare is a critical prerequisite 
for improvements to be achieved [1-3]. Among the three types 
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of healthcare quality measures, outcome measures are of 
great intrinsic interest because they reflect all of the aspects of 
the services provided, unlike structure or process indicators 
[4]. Mortality is one of the most commonly used outcome 
measures because the data is readily available and very useful, 
but robust risk adjustment methodologies are required if mor- 
tality is used as a quality measure. Disease-specific risk adjust- 
ment models for mortality prediction have been developed 
and are available for various diseases such as cardiovascular 
diseases and cancers. Although disease-specific risk adjust- 
ment is useful for quality improvement in certain clinical areas, 
generic risk-adjustment methodology that encompasses all 
clinical conditions and reflects the overall quality of a hospital 
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is more useful for hospital-wide quality improvement than 
disease-specific risk adjustment is. 

Several generic risk-adjustment measures such as the Me- 
disGroups and Disease Staging have been available for de- 
cades [5-7]. However, these generic measures require exten- 
sive efforts to collect detailed clinical data, thus are not widely 
used. In 1 999, the hospital standardized mortality ratio (HSMR) 
was developed and is a generic risk-adjustment measure that 
can be calculated from administrative data [8]. After its intro- 
duction in the UK, the HSMR has been widely used in many 
countries including the US, Canada, and the Netherlands [9]. 

In Korea, disease-specific risk-adjusted mortality has been 
evaluated nationally for various clinical conditions such acute 
myocardial infarction and major cancers since 2005. However, 
to the best of our knowledge, hospital-wide overall mortality 
has never been assessed using adequate risk-adjustment 
methodology. Thus, we sought to measure and compare hos- 
pital inpatient mortality among Korean hospitals using HSMR 
methodology. Specifically, the aim of this study was to assess 
variations in hospital HSMRs overall and to compare hospital 
HSMRs across hospital characteristics and disease groups. 

METHODS 

Study Population 

All inpatients who were discharged from any tertiary or gen- 
eral hospital in Korea with more than 700 beds between Janu- 
ary 1 and December 31 in 2008 were eligible for this study. We 
chose 700-bed hospitals as the cutoff point for inclusion to in- 
crease the accuracy of the measured mortality rates using an 
adequate sample size. Sixty-six hospitals fit this description, 
and more than half of all Koreans admitted to a hospital have 
visited one of these hospitals. 

Using the inpatient claims data from the National Health In- 
surance and Medical Aid from 2008, inpatients with conditions 
accounting for 80% of all inpatient deaths at these hospitals 
were selected. Our final study population included 703 571 pa- 
tients, which includes 27 718 (3.94% of all patients) in-hospital 
deaths nested in 66 hospitals (Table 1). We excluded deaths 
due to respiratory failure, cardiac arrest, and ventricular fibrilla- 
tion from the analysis because these deaths cannot be attrib- 
uted to poor quality health care rather than disease character- 
istics. We also excluded patients who were transferred in and/ 
or out of the hospital, those who were more than 100 years 
old, those who were less than 28 days old (neonates), and 



those who had a length of stay of less than one day or more 
than 365 days [9-16]. Patients admitted for palliative therapy 
were also excluded because their mortality correlates poorly 
with the quality of care received at the hospital. Patients who 
had inpatient claims within 7 days before or after the index ad- 
mission were defined as transferred patients. We excluded 
transferred patients because they tended to be sicker than 
non-transferred patients were. According to these exclusion 
criteria, 2884 patients were excluded from the study popula- 
tion (Supplemental Figure 1 ). 

Variables 

The patient outcome (alive and dead) and dates of death (if 
relevant), were obtained from the 2009 National Health Insur- 
ance registry. The patient characteristics required for calculating 
HSMRs were derived from the National Health Insurance and 
Medical Aid inpatient claims data. The claims data include pa- 
tient demographic information such as age, gender, principal 
diagnosis, comorbidities, surgical procedures, and the whether 
admission was elective or emergency. Patient comorbidities 
were abstracted from secondary diagnoses codes. In addition, 
hospital characteristics such as academic status (academic vs. 
non-academic), ownership (public vs. private), bed size, and 
geographic location (metropolitan vs. non-metropolitan) were 
identified. 

Hospital Standardized Mortality Ratio 

The final outcome variable in this study was the HSMR, 
which represents hospital-wide inpatient mortality. The HSMR 
is the ratio of the actual number of in-hospital deaths to the 
expected number of in-hospital deaths, for the most responsi- 
ble diagnosis (MRDx) accounting for 80% of inpatient mortality 
[16]. Patients were grouped according to their MRDx category 
using the HSMR methodology, and the Korean Classification of 
Diseases, 6th revision was used to map their principal diagno- 
ses to MRDx. Among 258 MRDx of all inpatient, 29 MRDx ac- 
counted for 80% of all deaths in this study. 

Using logistic regression, risk-adjusted mortality prediction 
models were built for each MRDx. Then, using model coeffi- 
cients, the number of expected deaths was determined based 
on the sum of the probabilities of in-hospital death for patients 
from each hospital. The logistic regression models incorporated 
the factors affecting mortality, including age, gender, urgency 
of admission, income level (National Health Insurance vs. Medi- 
cal Aid), and comorbidities. The Elixhauser comorbidity index 
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Table 1. Number of cases, fatalities, and the crude mortality rate for each MRDx 


MRDx group 


MRDx 


Cases (n) 


Deaths (n) 


Crude mortality rate (%) 


Certain infectious and parasitic diseases (A00-B99) 


Septicemia 


6720 


1832 


27.26 




Cancer of the breast 


46 669 


541 


1.16 




Cancer of the bronchus, lung 


64 328 


3436 


5.34 




Cancer of the colon 


48 853 


874 


1.79 




Cancer of the esophagus 


7674 


327 


4.26 




Cancer of the head and neck 


11 405 


286 


2.51 




Cancer of the liver and intrahepatic bile duct 


54 146 


3443 


6.36 


Neoplasms (C00-D48) 


Cancer of another Gl organ, peritoneum 


13 128 


890 


6.78 




Cancer of the pancreas 


11 848 


906 


7.65 




Cancer of the rectum & anus 


34 417 


557 


1.62 




Cancer of the stomach 


74 002 


2292 


3.10 




Leukemia 


16 051 


951 


5.92 




Non-Hodgkin's lymphoma 


16 355 


549 


3.36 




Secondary malignancy 


5794 


271 


4.68 




Aspiration pneumonia 


2561 


401 


15.66 


Diseases of the respiratory system (J00-J99) 


Chronic obstructive pulmonary disease 


12 832 


443 


3.45 




Other lower respiratory disease 


8310 


351 


4.22 




Pneumonia 


53 149 


1447 


2.72 




Acute cerebrovascular disease 


46 938 


2675 


5.70 


Diseases of the circulatory system (100-199) 


Acute myocardial infarction 


14 240 


883 


6.20 




Congestive heart failure 


8021 


402 


5.01 




LiUIUIIdly d LI lc[ UbU cl Ublb 


1 U J J 1 




n A7 


Diseases of the digestive system (K00-K93) 


Alcohol-related liver disease 


8194 


412 


5.03 




Gastrointestinal bleeding 


13 974 


400 


2.86 




Non-alcoholic liver disease 


16 031 


794 


4.95 


Diseases of the genitourinary system (N00-N99) 


Acute renal failure 


3275 


342 


10.44 




Chronic renal failure 


20 045 


640 


3.19 


Injury, poisoning, and other consequences of 


Intracranial injury 


10178 


636 


6.25 


external causes (S00-T98) 


Poisoning 


4082 


443 


10.85 




Total 


703 571 


27 718 


3.94 



MRDx, most responsible diagnosis; Gl, gastrointestinal. 



was used to adjust for comorbidities [17-20]. The diagnosis re- 
lated group (DRG) screening method embedded in the Elix- 
hauser comorbidity index was used to differentiate comorbid 
conditions present on admission from complications that oc- 
curred during hospitalization. If the MRDx was a malignant neo- 
plasm, variables reflecting cancer stage such as undergoing sur- 
gery, chemotherapy, and/or radiation were also included in the 
models. We also included principal diagnoses codes and dis- 
ease-specific mortality risk factors for further risk adjustment. 
The predictive power of the model was assessed using the c- 
statistic and the fitness of the model was assessed using the 
Hosmer-Lemeshow (H-L) statistic [21]. To avoid over-fitting the 
model, the bootstrapping technique was applied to each mod- 



el. In addition, 95% confidence intervals of the HSMRs were cal- 
culated using Byar's approximation [22]. Further information 
about our risk-adjustment model can be found elsewhere [23]. 

Statistical Analyses 

Variations in the hospital HSMRs across the included hospi- 
tals were assessed and their statistical significance was com- 
pared with the average HSMR, which is 100. In addition, multi- 
ple regression models were used to measure associations with 
hospital characteristics. To evaluate variations in hospital HSMR 
across different clinical conditions, we categorized hospitals 
into tertiles based on their overall HSMRs and the HSMRs of 
each MRDx group. Then, the HSMR distribution for each MRDx 
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group across each tertile was analyzed within the tertile of the 
overall HSMR. Using the 6th edition of the Korean Classification 
of Diseases system, 29 MRDx were categorized into the follow- 
ing seven MRDx groups: particular infectious and parasitic dis- 
eases (A00-B99), neoplasms (C00-D48), and diseases of the cir- 
culatory system (100-199), respiratory system (J00-J99), diges- 
tive system (K00-K93), and genitourinary system (N00-N99) as 
well as injury, poisoning, and certain other consequences of 
external events (S00-T98). The institutional review board of 
Seoul National University College of Medicine (H-0910-015- 
296) approved this study. All data analyses were performed us- 
ing SAS version 9.3 (SAS Inc., Cary, NC, USA). 



RESULTS 

Validity of Risk-adjusted Mortality Prediction 
Model 

Table 2 shows the c-statistics and H-L statistics from the lo- 
gistic regression models across the 29 MRDx. Except for the 
mortality prediction models of three MRDx, all of the models 
had c-statistics higher than 0.8. The c statistic was over 0.9 for 
five MRDx (17.2%) that included colon cancer, breast cancer, 
and pneumonia; between 0.9 and 0.8 in 21 MRDx (72.4%) that 
included acute cerebrovascular disease, acute myocardial in- 
farction, and acute renal failure; and less than 0.8 in the remain- 
ing three MRDx (10.3%) that included poisoning, aspiration 



Table 2. C-statistics and H-L statistics for each MRDx 



C-statistic 
range 


ivinuA 


Udaco \ll / 


-7lnnl 


U old 11 oil u 


95% CI of c-statistic 
Lower Upper 


H-L 

Pr>ChiSq 


C>0.9 


Cancer of the colon 


48 853 


5590.687 


0.938 


0.93 


0.94 


0.001 




Cancer of the breast 


46 669 


3739.008 


0.936 


0.93 


0.94 


<0.001 




Pneumonia 


53 149 


8989.760 


0.921 


0.92 


0.93 


0.001 




Cancer of the rectum and anus 


34 417 


3990.594 


0.915 


0.90 


0.93 


0.05 




Cancer of the stomach 


74 002 


14416.941 


0.903 


0.88 


0.93 


<0.001 




Gastrointestinal bleeding 


13 974 


2416.709 


0.900 


0.88 


0.91 


0.47 


0.8<C<0.9 


Cancer of the bronchus, or lung 


64 328 


20 960.620 


0.891 


0.87 


0.91 


<0.001 




Cancer of the head and neck 


11 405 


2004.884 


0.888 


0.86 


0.91 


0.14 




Acute myocardial infarction 


14 240 


4559.415 


0.886 


0.85 


0.92 


<0.001 




Other lower respiratory tract disease 


8310 


2092.522 


0.879 


0.86 


0.90 


0.09 




Coronary atherosclerosis 


70 351 


2906.599 


0.875 


0.84 


0.91 


0.08 




Septicemia 


6720 


5082.639 


0.872 


0.86 


0.88 


0.03 




Non-Hodgkin's lymphoma 


16 355 


3618.164 


0.869 


0.85 


0.88 


<0.001 




Cancer of the liver or the intrahepatic bile duct 


54146 


19 035.828 


0.868 


0.86 


0.87 


<0.001 




Cancer of the esophagus 


7674 


2102.728 


0.854 


0.83 


0.87 


0.11 




Non-alcoholic liver disease 


16 031 


4881.563 


0.845 


0.83 


0.86 


0.10 




Leukemia 


16 051 


5619.912 


0.843 


0.79 


0.90 


0.001 




Chronic obstructive pulmonary disease 


12 832 


3107.190 


0.842 


0.83 


0.86 


0.04 




Cancer of the pancreas 


11 848 


5026.615 


0.836 


0.82 


0.85 


0.06 




Secondary malignancy 


5794 


1747.959 


0.831 


0.81 


0.87 


0.06 




Acute renal failure 


3275 


1727.085 


0.829 


0.80 


0.85 


0.02 




Chronic renal failure 


20 045 


4692.210 


0.828 


0.81 


0.84 


0.002 




Intracranial injury 


10178 


3855.706 


0.821 


0.80 


0.84 


0.13 




Alcohol-related liver disease 


8194 


2708.256 


0.820 


0.80 


0.84 


0.24 




Cancer of another Gl organ, or the peritoneum 


13 128 


5286.592 


0.819 


0.81 


0.83 


0.20 




Acute cerebrovascular disease 


46 938 


17 024.397 


0.809 


0.80 


0.82 


0.01 


C<0.8 


Poisoning 

Aspiration pneumonia 
Congestive heart failure 


4082 
2561 
8021 


2361.975 
1965.775 
2947.031 


0.781 
0.732 
0.706 


0.75 
0.71 
0.67 


0.81 
0.76 
0.74 


0.36 
0.98 
0.47 



H-L, Hosmer-Lemeshow; MRDx, most responsible diagnosis; CI, confidence interval; Gl, gastrointestinal. 
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Mean 108.51 
Minimum 46.0 
Maximum 189.0 
Interquartile range 124.95 




Mean line 



Table 3. The hospital standardized mortality ratio for various 
hospital characteristics 



Each hospital 

Figure 1. Caterpillar plot of the overall hospital standardized mortality ratio (HSMR) for large hospitals in 2008. Each hospital's 
HSMR is graphed as the upper and lower 95% confidence limits. mM Hospitals had a significantly high HSMR, A24 hospitals had 
a non-significant HSMR, »25 hospitals had significantly low HSMR. 

pneumonia, and congestive heart failure. However, in 14 out of 
29 MRDx, the H-L statistics of these models were significant 
(p < 0.05), which suggests poor model fitness. 

Overall Variation Among Each Hospital's Hospital 
Standardized Mortality Ratio 

The HSMR widely varied across the hospitals included in this 
study (Figure 1). Of all of the hospitals, 25 (37.9%) had a signif- 
icantly low HSMR and 17 hospitals (25.8%) had a significantly 
high HSMR. The remaining 24 hospitals (36.4%) showed an av- 
erage HSMR. The highest hospital HSMR (189.0) was 4.1 times 
higher than the lowest HSMR was (46.0). 

Variation Across Hospital Characteristics 

The HSMR for different sub-groups of hospitals are presented 
in Table 3 and Supplemental Table 1 .The academic status of the 
hospital and Medical Aid insurance status of the patients were 
significantly associated with HSMR; however, the other hospital 
characteristics such as hospital location, the number of doctors, 
having older age patients, having an emergency admission, 
having undergone surgical procedures, and having cancer pa- 
tients were not related. Moreover, large variations were found 
among non-academic hospitals (vs. academic), public hospitals 
(vs. private hospitals), and hospitals with less than 800 beds (vs. 
those with 800 beds or more). 



Variation Across Most Responsible Diagnosis 
Groups 

To evaluate agreement between overall HSMR and disease- 



Variables 


n 


Mean 


SD 


Minimum 


Maximum 


p-value 1 


Total, all hospitals 


66 


108.5 


27.1 


46 


189 


- 


Academic status 












- 


Academic 


56 


104 


23.8 


46 


154.9 


0.001 


Non-academic 


10 


133.7 


32 


73.4 


189 




Ownership 














Public 


12 


106 


36 


59.1 


189 


0.73 


Private 


54 


109.1 


25.1 


46 


167.5 




Beds 














<800 


18 


114 


33.2 


73.4 


189 


0.20 


800-1000 


29 


111.2 


22.7 


70.9 


154.9 




>1000 


19 


99.2 


26.2 


46 


151.1 




Hospital location 














Metropolitan 


42 


109.6 


27.8 


46 


189 


0.66 


Non-metropolitan 


24 


106.6 


26.5 


66.5 


154.9 




No. of doctors 














per 100 beds 














<20 


4 


133.3 


48.6 


73.4 


189 


0.05 


20-30 


10 


114.7 


32.2 


75.6 


167.5 




30-40 


27 


101.4 


23.5 


46 


154.9 




40-50 


17 


116.7 


21.1 


83.5 


151.1 




>50 


8 


95.3 


21.1 


59.1 


127.5 




Nursing grade 














1 


4 


81.2 


35.1 


46 


128.4 


0.13 


2 


14 


108.4 


27.1 


59.1 


154.9 




3 


30 


107.4 


24.1 


73.4 


167.5 




4+ 


18 


116.5 


28.3 


84.8 


189 





'All p-values were calculated using f-tests and AN0VA with the Bonferroni 
correction for multiple comparisons. 
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Table 4. The number of hospitals across the tertiles of the standardized mortality ratio for each MRDx group 





HSMR 
tertiles 1 


Overall HSMR tertiles 1 








MRDx group 


High 
(115 6-189 0) 


Middle 
(95 0-1151) 


— ; 

Low 

(46.0-94.6) 


Agreement 

(%) 


Kappa 


All 




22 


22 


22 






Certain infectious and parasitic diseases (A00-B99) 


High (109.0-167.8) 
Middle (89.5-108.5) 


12 

c 

3 


6 

q 


4 
8 


47.0 


0.205* 




Low (47.7-89.3) 


J 


7 
/ 


10 






Neoplasms (C00-D48) 


High (130.6-259.9) 


1 R 


A 
4 


0 


78.8 


0.682*** 




Middle (97.2-130.4) 


A 
4 


I J 


3 








Low (44.1-93.3) 


n 
u 


0 


19 






Diseases of the circulatory system (100-199) 


High (114.2-168.5) 
Middle (93.8-1 14.0) 
Low (43.7-92.9) 


i j 

Q 
O 

1 
1 


c 
D 

q 
7 


3 
5 

14 


54.5 


0.318*** 


Diseases of the respiratory system (J00-J99) 


High (120.1-200.9) 


1 A 
I 4 


0 


3 


54.5 


0.318*** 




Middle (93.2-1 18.4) 


7 


q 


6 








Low (32.5-91.9) 


1 


8 


13 






Diseases of the digestive system (K00-K93) 


High (120.2-242.3) 


14 


5 


3 


62.1 


0.432*** 




Middle (85.8-1 19.8) 


5 


12 


4 








Low (41.0-84.9) 


2 


5 


15 






Diseases of the genitourinary system (N00-N99) 


High (118.1-295.0) 


13 


8 


1 


49.2 


0.239** 




Middle (82.9-1 15.7) 


6 


7 


9 








Low 2 (11.4 -80.6) 


3 


6 


12 






Injury, poisoning and other consequences of 
external causes (S00-T98) 


High (110.6-206.6) 
Middle (79.5-109.8) 
Low 3 (12.3-78.3) 


12 
5 
4 


CO CO CD 


4 
9 
8 


43.8 


0.156 



MRDx, most responsible diagnosis; HSMR, hospital standardized mortality ratio. 
'Number in parenthesis refers to the HSMR range for each HSMR tertile. 
2 0ne hospital experienced no deaths and was excluded from the HSMR calculation. 
Two hospitals experienced no deaths and were excluded from the HSMR calculation. 
*p<0.05, **p<0.01,***p<0.001. 



specific HSMR among hospitals, hospitals were categorized 
into tertiles according to overall HSMR and HSMR by MRDx 
groups. The agreement between overall HSMR and disease- 
specific HSMR was limited (Table 4). In four out of seven MRDx 
groups, the MRDx group-specific HSMR showed fair agree- 
ment (kappa, 0.21 to 0.40) with the overall HSMR regarding 
the strength of the agreement. The level of agreement be- 
tween the overall HSMR and HSMR by MRDx was fair and was 
highest among those with neoplasms (kappa, 0.682), yet low- 
est among those with injury, poisoning, and other external 
causes (kappa, 0.156). 

DISCUSSION 

We found that the hospital HSMR among large hospitals in 
Korea varied greatly. The mortality of the highest HSMR was 4.1 



times higher than that of the lowest HSMR. Out of the 66 hos- 
pitals, about two-thirds of them showed either significantly 
high or low mortality while only about one-third showed aver- 
age mortality. The magnitude of variations in hospital HSMR 
was larger than those of previous studies were. The range of 
the HSMR in this study was larger (46 to 1 89) than that in the 
UK (53 to 137 in 1999), Canada (56 to 142 from 2004 to 2007), 
and the Netherlands (68 to 1 32 in 2009) [1 0,1 5,24,25]. We con- 
sidered the HSMR of large hospitals in Korea only; therefore, 
the HSMR variation among Korean hospitals could be larger if 
all hospitals are compared. Hospitals with a high mortality rate 
should make efforts to reduce it, and at the national level, ef- 
forts should be made to decrease the large variation in the 
mortality rates of all hospitals. 

Among hospital characteristics included in the analysis, the 
academic status of the hospital and Medical Aid insurance sta- 
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tus of the patients were associated with hospital HSMR. This 
finding was consistent with those of previous studies. Teaching 
hospitals showed lower hospital mortality than non-teaching 
hospitals did [26-28]. In addition, Medical Aid patients general- 
ly have higher mortality rates than non-Medical Aid patients 
do [29,30]. However, hospital location, number of doctors, hav- 
ing older age patients, emergency admission, undergoing a 
surgical procedure, and having cancer patients were not associ- 
ated with hospital HSMR. Nevertheless, hospital characteristics 
might differ in their association with hospital HSMR if small 
hospitals are added to the analysis. The hospital HSMR also 
showed a fair level of agreement in most MRDx groups; there- 
fore, a hospital showing a low mortality for certain clinical con- 
ditions did not necessarily have low mortalities for other clini- 
cal conditions. For example, hospitals that had a low mortality 
for neoplasm tended to have a high mortality in other condi- 
tions such as digestive diseases. This finding indicates that 
there are disease-specific factors determining mortality in ad- 
dition to generic factors and that disease-specific hospital mor- 
tality should be evaluated with overall hospital mortality. The 
limited agreement of hospital HSMR across the MRDx groups 
also suggests that disease group-specific mortality should be 
disclosed to facilitate targeted quality improvement effort in 
hospitals and informed choices among patients. 

The c statistic and H-L statistic are generally used to evaluate 
the predictive power and robustness of risk-adjusted models. 
The predictive power of our mortality prediction models was 
high. For most models, the c statistic had excellent (0.8 to 0.9) 
or outstanding (>0.9) discriminative value while only three 
models showed acceptable discriminative value (0.7 to 0.8). 
The prediction models of this study also had a better range of c 
statistics (0.706 to 0.938) than previous studies did. In the UK 
and the Netherlands, the range was 0.668 to 0.939 [31] and 
0.677 to 0.957 [32], respectively. Approximately half of all mor- 
tality prediction models indicated poor fitness according to the 
H-L statistic. However, when mortality prediction models were 
rebuilt using a subset of the study population for each MRDx, 
the H-L statistics of the models were no longer significant ex- 
cept for three MRDx models. This finding suggests that the sig- 
nificant H-L statistics of these models might have resulted from 
our large sample size rather than from poor fitness. 

This study has several limitations regarding use of claims 
data for risk adjustment. First, claims data may not be accurate 
enough for risk-adjusted mortality prediction because code 
creep in claims data has been known to be prevalent. Thus, to 



evaluate the accuracy of our claims data, we chose two MRDx 
(pneumonia and colon cancer), and then compared comorbidi- 
ties obtained from the medical records of all inpatients with 
these diseases who were admitted at four randomly selected 
hospitals in our study. We found that only 6% of comorbid con- 
ditions that appeared in the claims data were not found in the 
medical records. Second, the DRG screening method embed- 
ded in the Elixhauser comorbidity index may not accurately 
differentiate comorbid conditions from complications that oc- 
curred during hospitalization. Thus, to evaluate the accuracy of 
the DRG screening method, we compared the results of the 
DRG screening with that of the medical records we reviewed 
using the inpatient episode to evaluate the accuracy of the 
claims data. We found that only 6% of comorbid conditions 
based on DRG screening were judged as complications accord- 
ing to the medical records. The rate of disagreement was 4.1 % 
for pneumonia and 23.4% for colon cancer. However, to differ- 
entiate comorbid conditions from complications accurately, the 
diagnosis coding system for claims data should include flags to 
discern whether a disease developed before or after hospital- 
ization [33]. Third, the predictive power of mortality prediction 
models using claims data may not be as robust as those made 
using clinical data are. Thus, to compare predictive power of 
the claim data based models and clinical data based models, c- 
statistics of the claim data based model and clinical based 
model (using the inpatient data) were calculated. The model 
based on clinical data included comprehensive riskfactors such 
as vital signs (temperature, blood pressure, and respiration 
rate) and laboratory results (pH, hemoglobin, and creatinine). 
The c-statistic of the model based on clinical data for pneumo- 
nia was 0.005 higher than that of the model based on claims 
data (0.940 vs. 0.945). Because of the small sample size for this 
analysis, a model based on clinical data that incorporated all 
variables of the model based on claims data could not have 
been built. 

This study also has several methodological limitations, which 
are common in studies using HSMR methodology. First, coding 
variations among hospitals can result in inappropriate risk ad- 
justment [1 0,1 4,33]. Because the comorbidity index and emer- 
gency admission episodes are two important variables for risk 
adjustment in HSMR methodology, variations in coding prac- 
tice for these variables may have a potentially large impact on 
the HSMR of individual hospitals. When we included the inter- 
action term of the Elixhauser comorbidity index for each hospi- 
tal and individual hospital in our mortality prediction models, 
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the interaction terms were significant only in 20 hospitals out 
of the 66 total hospitals. In addition, the odds ratios were not 
large (0.71 3 to 1 .382) across these hospitals with significant in- 
teraction terms. Hospitals that tend to treat emergency pa- 
tients with mild symptoms may have a lower HSMR than those 
that tend to treat more severe emergency patients, regardless 
of their quality of care [14]. In this study, the emergency admis- 
sion and number of transfer episodes with a 1-day length of 
stay had a low mortality of 2.1 9% compared to the overall mor- 
tality. Therefore, we included only patients who visited the 
emergency room if they were charged a fee for emergency 
medical service and had a length of stay longer than 1 day. By 
clearly those who visited the emergency room, we tried to 
minimize the potential impact of different case-mix of these 
emergency patients on the HSMR calculation of individual hos- 
pitals. Second, applying different mortality timeframes may re- 
sult in discrepancies between the HSMR of individual hospitals, 
and previous studies have reported inconsistent findings re- 
garding the consequences of applying different mortality time- 
frames on HSMR [34,35]. We used in-hospital mortality as the 
mortality timeframe because other countries have reported 
national HSMR using in-hospital mortality as the timeframe [9], 
and the c-statistics of the mortality prediction models used in 
calculating the in-hospital mortality timeframe were higher 
than those employing a different mortality timeframe were. 
Moreover, a patient's cause of death post-discharge could have 
been unrelated to the patient's reason for previous admission. 
Third, the exclusion criteria of the HSMR methodology might 
create a selection bias [33]. We excluded transferred patients, 
admissions with an extremely long length of stay, and patients 
older than 100 years old from the analysis. However, in our 
study, excluding these patients did not seem to cause a sub- 
stantial selection bias because it required only approximately 
0.4% of the study population (2884 out of 706 455) to be ex- 
cluded from the analysis. 
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Study population: 706 455 patients 
in 66 hospotals with more 
than 700 beds in 2008 
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Supplemental Figure 1. Schematic representation of the study population selection an analysis process. LOS, length of stay; 
HSMR, hospital standardized mortality ratio. 



Supplemental Table 1. Results of the multiple regression 
analysis for the relationship between each hospital character- 
istic and the hospital standardized mortality ratio 



Variables 


P 


t 


p-value 


Intercept 


193.637 


4.31 


<0.001 


Academic status 








Academic 


-27.093 


-2.14 


0.04 


Non-academic 


0 






Hospital location 








Metropolitan 


3.83 


0.56 


0.58 


Non-metropolitan 


0 






No. of doctors per 1 00 beds 








< 20 


-22.17 


-1.05 


0.30 


20-30 


-3.758 


-0.26 


0.79 


30-40 


-6.231 


-0.58 


0.56 


40-50 


17.254 


1.56 


0.12 


>50 


0 






Over 65 years old (%) 


-0.494 


-0.7 


0.49 


Recipient of Medical Aid (%) 


1.863 


2.02 


0.05 


Admitted via the ER(%) 


-0.653 


-1.59 


0.12 


Received surgery (%) 


-0.775 


-1.31 


0.19 


Cancer diagnosis (%) 


-0.456 


-1.73 


0.09 






0.380 





ER, emergency room. 



Excluded: 2884 patients 

- Deaths from respiratory failure, cardiac arrest, ventricular fibrillation 

- Patients who were more than 1 00 years old and neonates 

- Patients with LOS of under 1 day or over 365 days 

- Patients who were transferres in and out 

- Palliative therapy patients 
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